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Consider M-estimation in a semiparametric model that is charac- 
terized by a Euclidean parameter of interest and a nuisance function 
parameter. We show that, under general conditions, the bootstrap 
is asymptotically consistent in estimating the distribution of the M- 
estimate of Euclidean parameter; that is, the bootstrap distribution 
asymptotically imitates the distribution of the M-estimate. We also 
show that the bootstrap confidence set has the asymptotically correct 
coverage probability. These general conclusions hold, in particular, 
when the nuisance parameter is not estimable at root-n rate. Our 
results provide a theoretical justification for the use of bootstrap as 
an inference tool in semiparametric modelling and apply to a broad 
class of bootstrap methods with exchangeable bootstrap weights. A 
by-product of our theoretical development is the second order asymp- 
totic linear expansion of the (bootstrap) M-estimate. 



1. Introdution. Due to its flexibility, semiparametric modelling has 



provided a powerful statistical modelling framework for complex data, and 



proven to be useful in a variety of contexts, see 



Q,y, 



13, 



25 



26 



,y,Q,Q]. 
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Semiparametric models are indexed by a Euclidean parameter of interest 
6* G e C M"' and a nuisance function parameter t] belonging to a Ba- 
nach space 7i with norm || • ||. M-estimation, including the maximum likeli- 
hood estimation as a special case, refers to a general method of estimation, 
lere the estimates are obtained by optimizing certain criterion functions 



w. 

B 



15,^ 



38| . The asymptotic theories and inference procedures for semi- 



parametric maximum likelihood estimation, or more generally M-estimation 



have been extensively studied in 



m 



27 



28|, 



36| . In particular, a gen- 



eral theorem for investigating the asymptotic behavior of M-estimate for 6 
in semiparametric models is given in [381 . 

The bootstrap method is a widely used data-resampling method in draw- 
ing statistical inference such as obtainig standard errors and constructing 



confidence regions. See 



14 



18|, 



22 



24 



33( 1 for its application in semi- 



parametric models. By replacing complicated theoretical derivations with 
the routine simulation of bootstrap samples, the bootstrap method is con- 
ceptually and operationally simple. It is such simplicity that leads to the 
popularity of the bootstrap method. However, the validity of the bootstrap, 
i.e., why it yields the valid inference, needs to be theoretically justified. 
While the asymptotic validity of the bootstrap method has been well es- 



tablished in parametric models 



13|, 



32l |. a systematic theoretical study on 



the bootstrap inference in semiparametric models is almost non-existence, 
especially when the nuisance function rj is not y/n convergent. This paper 
tries to fill in this gap. 

An unfortunate observation is that even the basic bootstrap consistency 
theorem has never been established for semiparametric models. The lack 
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of theoretical justifications of the bootstrap in the semiparametric context 
leads to the development of other semiparametric inferential tools, e.g., pig- 



gyback bootstrap 



2l[ |. Although having established 



9(1 and profile sampler 
asymptotic theory, these alternatives of the bootstrap have their own short- 
comings. The piggyback bootstrap is not the standard bootstrap and re- 
quires knowing the limiting distribution of MLE for 9, which may not be 
easy to estimate in general. The implementation of profile sampler involves 
the computation of the profile likelihood and may have some MCMC com- 
putational issues, e.g. the choice of prior and the convergence of the Markov 
chain. Moreover, neither of these methods is as simple to implement as the 
classical bootstrap of Efron. The purpose of this paper is to develop a general 
theory on bootstrap consistency for the classical bootstrap method and thus 
promote its use as an inference tool for semiparametric models. We focus 
on the Euclidean parameter of the model. Our main results are summarized 
below. 

The semiparametric M-estimator {6, rj) and bootstrap M-estimator {6*,r]*) 
are obtained by optimizing the objective function m{6, rj) based on the i.i.d. 
observations (Xi, . . . , Xn) and bootstrap sample {XI, . . . , X*), respectively: 

n 

(1) {e,r])=aig sup Vm(6',r/)(Xi), 

n 

(2) (r,7y*)=arg sup J] m(e, 7y)(X*). 
Note that we can also express 

n 

(3) (r,r) = arg sup ^ T^™m(e, 7?)(X,). 

by using the bootstrap weights Wni^s. For example, if {XI, . . . ,X*) are in- 
dependent draws with replacement from the original sample, i.e. Efron's 
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(Nonparametric) Bootstrap, then Wn = {Wni, ■ ■ ■ ,Wnn) follows the Multi- 
nomial distribution with parameter (n, (n~^, ■ ■ ■ ,n~^)). The more general 



29( 1 that includes Efron's bootstrap 



exchangeable bootstrap weighting scheme 
as a special case are considered in this paper. 

We first present a preliminary result showing the asymptotic normality of 
semiparametric M-estimate 6 in Theorem [TJ As a key result in this paper, 
our Theorem [2] confirms that this limiting distribution can be bootstrapped 
consistently. For example, in the Efron's bootstrap, the bootstrap distribu- 
tion of ^/n{9* — 9), conditional on the observed data, asymptotically imitates 
the distribution of y/n{9 — 9Q), where is the true value of 9, see CorollarylH 
As shown in Theorem[3l the consistency of the bootstrap confidence set of 9, 
which means that its coverage probability converges to the nominal level, im- 
mediately follows from the above distributional consistency theorem. All the 
above conclusions are valid, in particular, when the nuisance parameter has 
slower than root-n convergence rate. The rigorous proof of Theorem[2]is very 
challenging since it requires careful probabilistic analysis, see Lemma lA.H 
and involves the bootstrapped empirical processes techniques. 

As a by-product of our theoretical development, we also obtain the second 
order asymptotic linear expansions for the M-estimator of the Euclidean 
parameter and for its bootstrap version, which imply that the second order 
accuracy of the (bootstrap) M-estimate depends on the convergence rate 
of the nuisance function parameter. Such results extend similar results of 



Cheng and Kosorok |7| for MLE and are of its own interests. 

In a highly related paper. Ma and Kosorok (2005) obtained some consis- 
tency results for what they called the weighted bootstrap where the boot- 
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strap weights are assumed to be independent. This independence 

assumption rules out the possiblity of applying their results to justify Efron's 
boostrap in the semiparametric setting. They stated in the paper that the 
independence assumption makes the proofs easier and the relaxation to the 
dependent weights appears to be quite difficult. In this paper, we make use of 
the bootstrapped empirical processes techniques [l^ to overcome the techni- 
cal difficulties. Other related work includes interesting results on bootstrap 
consistency in nonparametric models. Wellner and Zhan (1996) proved the 
bootstrap consistency for root-n convergent nonparametric estimates ob- 
tained by solving estimation equations. Sen et. al. (2009) and Kosorok (2008) 
showed the inconsistency of bootstrapping Grenander estimator. 

Our main results, including the bootstrap consistency theorem, are pre- 
sented in Section 2. Sections 3 and 4 discuss how to verify various technical 
conditions needed for the main results. Section 5 illustrates the applications 
of our main results using three examples. Sections 6 contains proofs of our 
main results in Section 2. Some useful lemmas and additional proofs are 
postponed to Appendix. 

2. Main Results. 

2.1. Preliminaries. We first introduce a paradigm for studying the semi- 
parametric M-estimate 6, which parallels the efficient influence function 
paradigm used for MLEs (where m{6,i]) = log lik{6,i])), developed in Sec- 



tion 6 of 



33]. Let 



d d 
mi{9,r]) = —m{e,r]) and m2{9,r])[h] = —\t=om{e,r]{t)), 
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where h is a "direction" along which ri{t) G TC approaches as t — > 0, 
running through some index set H C L^[Pq^^). Similarly we also define 

d d 
mu{9,rj) = —mi{e,r]) and mi2{9,rj)[h] = —\t=omi{e,r]{t)), 

d d 
m2i{9,ri)[h] = —m2{9,r])[h] and 17122(9, rj)[h, g] = —\t=om2{9,r]2{t))[h], 

where h,g £11 and (d / dt)\t-Qri2(t) = g. Define 

m2{9,ri)[H] = {m2{9,r,)[hi], . . . ,m2{9,r,)[hd]y 

m22[H,h] = {m22{0,r])[hi,h],..., 17122(9, r])[hd,h]y, 

where H = (hi, . . . , h^) and hj G H for j = 1, . . . ,d. Assume there exists an 

H\9,rj) = (hl(9,r^),...,hl(9,r^)y, 

where each hj(9,r]) G H, such that for any /i G H 

(4) Ee, ^{mi2(9,r])[h]- 77122(9, r])[H^, h]} =0. 

Following the idea of the efficient score function, we define the function 
fh(9, rf) = mi(9, rf) — m2(9, r])[H^(9, r/)]. 
Based on the constructions oimi(9,ri) and m2(9,r])[H], (9,rj) satisfies 

(5) P„m(0,r?) = O, 

where P^/ denotes X^iLi /(^«)/^- We assume that the observed data are 
from the probability space (X,A,Px)-, and that 

(6) Pxm(9o,7]Q) = Q, 
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where Pxf is the customary operator notation defined as J fdPx- The as- 



38( 1 and usu- 



sumption ([6]) is common in semiparametric M-estimation 
aUy holds by the semiparametric model specifications, e.g. the semipara- 
metric regression models with "panel count data" 



. In particular, when 



m{9, 7]) = log lik{6, t]), ([6|) trivially holds and fh{9, rf) becomes the well stud- 
ied efficient score function for 9 in semiparametric models, see Q]- 

Throughout the rest of the paper, we use the shortened notations ifg = 
{(^0,110)1 ^0 = f^i^^OiVo) ^-iid fh = fh{9,rf), and we use the superscript 
"o" to denote the outer probability. For a probability space {VL,A,P) and a 
map r : 17 M that need not be measurable, the notations E°T, Op^(l), 
and Op^(l) represent the outer expectation of T w.r.t. P, bounded and 
converges to zero in outer probability, respectively. More precise definitions 



can be found on Page 6 of 3^. Let represent the outer product matrix 
VV' for any vector V . Define x V y {x Ay) to be the maximum (minimum) 
value of X and y. 

We now state some general conditions that will be used throughout the 
whole paper. We assume that the true value 9o of the Euclidean parameter 
is an interior point of the compact set Q. 

I. Postive Information Condition: The matrices A = Px{'miii9o,r]o) ~ 
m2i{9o,r]o)[Hl]) and B = Px[{mii9o,Vo)-m2{9o,Vo)[Hl]}®^] are both 
nonsingular. 

Condition I above is used to ensure the nonsingularity of the asymptotic 
variance of 9 in ()14l) . which will be shown to be A~^B{A~^)' . 

For the empirical process G„ = -^/n(P„— -Px )> denote its norm with respect 
to a function class Tn as ||G„||jf„ = supjgjp^ For any fixed 5„ > 0, 
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8 GUANG CHENG AND JIANHUA Z. HUANG 

define a class of functions 5„ as 

(7) Sn = Sn[On) = < n H ' WV " ^0 < K 

I \\V-Vo\\ 
and a shrinking neighborhood of (^0)%) a-s 

(8) Cn = CniSn) = {{9, V) ■ P " ^o|| < Sn, \\v " m\\ < ^n} ■ 

The next two conditions, Conditions S1-S2, make sure that the empirical 
processes indexed by m{9, r]) are well behaved and fh{6, rf) is smooth enough 
around (^Oi 

51. Stochastic Equicontinuity Condition: for any 5n — > 0, 
(9) ||G„|k=0?,^(l) 

and 

{W)Gn{m{9,v)-M9o,v)) = 0°p^{\\e-eo\\) for {9,r])eCn. 

52. Smoothness Condition: 

(11) Px{m{e, 7]) - mo) = A{e - Oo) + 0{\\e - Oof V ||r? - r/of ), 
for {6,r]) in some neighborhood of {6(),r]Q). 

For any fixed 9, define 

rfg = arg sup P„m(0, r/). 

The next condition says that fje should be close to rjo if 9 is close to 9o. 

53. Convergence Rate Condition: there exists a 7 G (1/4, 1/2] such that, 

(12) \\ff^-r^^\\=0'},J\\e-9o\\Vn~^), 
for any consistent 9. 
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Here 7 is required to be larger than 1/4, which is always true for regular 
semiparametric models (see Section 3.4 of [s^). 

Verifications of Conditions S1-S3 will be discussed in Sections [3] and U] 
and illustrated with examples in Section O 

2.2. Semiparametric M-estimator. We show that the semiparametric M- 
estimator 9 has an asymptotic linear expression and it is asymptotically nor- 
mally distributed. This result plays an important role in proving bootstrap 
consistency in the next subsection. 

Theorem 1. Suppose that Conditions I, S1-S3 hold and that {9,rf) 
satisfies (0). // 9 is consistent, then 

(13) V^{9 - 9o) = -V^A-^Fnfho + Oj,^ (n-2^+V2). 
Thus, we have 

(14) V^{9-9o)^n{o,j:), 

where S = A^^ B(A~^y , A and B are given in Condition I. 

We assume consistency of 9 in Theorem[TJ The consistency can be usually 
guaranteed under the following "well-separated" condition: 



(15) Pxm{9o,'no)> sup Pxm{9,rf) 



35|. 



for any open set G C @ x H containing {9o,r]o), see Theorem 5.7 in 

To obtain the asymptotic normality, we only need the remainder term in 
the asymtotic linear expansion (jl3p to be of order Op^(l). Since 7 > 1/4, the 
order of the remainder term Op^{n~^"''^^^^) is always Op^(l). We actually 
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obtain a higher order expansion, stronger than what is needed for showing 
asymptotic normaUty. It is interesting to note that the rate of convergence 
of the remainder term depends on how accurately the nuisance function 
parameter rj can be estimated. In other words, we can conclude that the 
second order estimation accuracy of 9 is higher for semiparametric models 
with faster convergent 77. This higher order result extends a similar result 
of Cheng and Kosorok [a, lZ| developed for maximum likelihood to the M- 
estimation setting. 

For maximum likelihood estimation, ■m{6,r]) = loglik{6,r]), and it is easy 
to see that A = —B and S = B^^. In this case, becomes the efficient 
information matrix. 

Remark 1. Given any consistent estimator S o/S, we have 
(16) ^t~^'\d - 9o) ^ N{0, 1) 

by Theorem [7] and Slutsky 's Theorem. In practice, the consistent S can he 
obtained via either the observed profile information approach ^/ or the 



profile sampler approach 



Remark 2. The M- estimation equation can be relaxed to the 
"nearly maximizing" condition ¥nm{9,f]) = o°p^{n~^/'^). Under weaker con- 
ditions than stated in Theorem [II the same argument can be used to show 

(17) s/7i{B- 9o) = -V^A-^Fnmo + o?>^(l), 

which also implies the asymptotic normality of 9, i.e. ^14\ ) and il6\) . Note 



that |i7p has also been established in 123 . 



33 ]. but our argument can be used 



to obtain the higher order expansion result (jl3|) . 
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2.3. Bootstrap Consistency. In this subsection, we establish the consis- 
tency of bootstrapping 9 under general condtions in the framework of semi- 
parametric M-estimation. Define P*/ = (l/?^)I]"=i Wnif{Xi), where W„j's 
are the bootstrap weights defined on the probability space {W,Q, Pw)- In 
view of dS]), the bootstrap estimator can be rewritten as 

{6*,rj*) = avg sup '¥*^m{9,ri). 
eee,rien 

Similar to ([5]), we can show that {6*,r]*) satisfies 

(18) p;m(r,r?*) = 0. 

The bootstrap weights Wni^s are assumed to belong to the class of ex- 



29[. Specifically, they satisfy 



changeable bootstrap weights introduced in 
the following conditions: 

Wl. The vector Wn = iWni, ■ ■ ■ , Wnn)' is exchangeable for all n = 1,2, .. ., 
i.e. for any permutation vr = (tti, . . . , 7r„) of {1,2, ... ,n), the joint 
distribution of ^{Wn) = {Wnni, ■ ■ ■ , WmrnY is the same as that ofWn- 

W2. Wni > for all n, i and Y.i=i for all n. 

W3. limsup^^oc ll^^nilb,! < C < oo, where ||I^ni||2,i = /o°° \/Pw{Wni > u)di 
W4. limA„.oolimsup„^o^supi>;^t2Piy(W^«i > t) = 0. 
W5. (1/n) E7=i{Wm - If ^ c2 > 0. 

In Efron's nonparametric bootstrap, the bootstrap sample is drawn from 
the nonparametric estimate of the true distribution, i.e. empirical distribu- 
tion. Thus, it is easy to show that Wn ~ Multinomial(n, (n~^, . . . , n~^)) 
and Conditions W1-W5 are satisfied with c = 1 in W5. In general. Condi- 
tions W3-W5 are easily satisfied under some moment conditions on Wni, see 

imsart-aos ver. 2006/01/04 file: bconsistency-JH_v6.tex date: June 6, 2009 



12 GUANG CHENG AND JIANHUA Z. HUANG 

n 

Lemma 3.1 of [22i]. In addition to Efron's nonparametric boostrap, the sam- 
pling schemes that satisfy Conditions W1-W5 include Bayesian bootstrap, 



Multiplier bootstrap, Double bootstrap, and Urn boostrap; see 

There exist two sources of randomness for the bootstrapped quantity, e.g. 
9*: one comes from the observed data; another comes from the resampling 
done by the bootstrap, i.e. random WniS. Therefore, in order to rigorously 
state our theoretical results for the bootstrap, we need to specify relevant 
probability spaces and define stochastic orders with respect to relevant prob- 
ability measures. 

We view Xj as the ith coordinate projection from the canonical probability 
space {X°° ,A°° ^Px) onto the ith copy of X. For the joint randomness 
involved, the product probability space is defined as 

In this paper, we assume that the bootstrap weights W^i's are independent 
of the data Xj's, thus Pxw = Px ^ ^w- We write P^ as Px for simplicity 
thereafter. Define Exy/ as the outer expectation w.r.t. Pxw- The notations 
E°x and Ew are defined similarly. 
Given a real-valued function A„ defined on the above product probability 
space, e.g. 0* , we say that A„ is of an order Op^(l) in P^-probability if for 
any e, > 0, 

(19) PnPw\x{\^n\>e)>r,} 

as n ^ 0, and that A„ is of an order Op^(l) in P^-probability if for any 
r/ > 0, there exists a < Af < cxo such that 

(20) Pl{P^|^(|A„|>Af)>r?}^0 
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SEMIPARAMETRIC BOOTSTRAP CONSISTENCY 13 

as n — > oo. 

Given a function r„ defined only on , A"^ , Px), if it is of an order 
o°p^{l) (0|,^(1)), then it is also of an order o^^^(l) {0°p^^{l)) based on 
the following argument: 

P^wil^nl >e) = ^^v^Ulrnl > e} = ExEw\xH\rn\ > e}° 
= ^xi{|r„| >er = Pl{|r„| >e}, 

where the third equation holds since r„ does not depend on the bootstrap 
weight. More results on transition of various stochastic orders are given in 
Lemma lA.ll of the Appendix. Such results are used repeatedly in proving 
our bootstrap consistency theorem. 

To establish the bootstrap consistency, we need some additional condi- 
tions. The first condition is the measurability condition, denoted as M{Px)- 
We say a class of functions J- G M{Px) if J' possesses enough measur- 
ability so that P„ can be randomized, i.e. we can replace {5xi — Px) by 
iWni — ^)5xi^ and Fubini's Theorem can be used freely. The detailed de- 
scription for M{Px) is spelled out in [2] and also given in the Appendix of 
this paper. Define T = {m,{6,ri) : \\6 — 9q\\ + \\ri — r]Q\\ < R} for some R > 0. 
For the rest of the paper we assume T S M{Px)- 

The second class of conditions parallels Conditions S1-S3 used for ob- 
taining asymptotic normality of M-estimators and is only slightly stronger. 
Thus, the bootstrap consistency for the Euclidean parameter in semipara- 
metric models is almost automatically guaranteed once the semiparametric 
M-estimate is shown to be asymptotically normal. Let Sn{x) be the envelop 
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function of the class Sn = Sn{Sn) defined in ([7]), i.e. 

m{0o,r]) - fho 



Sn{x) = sup 

h-m\\<Sn 11^ -%l 

The next condition controls the tail of this envelop function. 
SBl. Tail Probability Condition: 

(21) lim lim sup supt'^P^{Sn{Xi) > t) = 

A— »oo n^oo t>A 

for any sequence 5„ — > 0. 

Let f = {dm{6,r])/de : {e,ri) G C„}, where C„ = C„((5„) is defined in 

SB2. We assume that f e M{Px) n L2{Px) and that f is P-Donsker. 

Condition SB2 ensures that the size of the function class T is reasonable 
so that the bootstrapped empirical processes G* = ^/^(P* — P„) indexed by 
las a limiting process conditional on the observations; see Theorem 2.2 



r 



m 



29]. 



For any fixed 9, define 

r/g = argmaxP*m(0, ry). 

The next condition says that rj^ should be close to rjo if 9 is close to ^o- 

SB3. Bootstrap Convergence Rate Condition: there exists a 7 G (1/4,1/2] 
such that 

(22) \\f}t-r]o\\=0"p^^.{\\9-6o\\V7i-^) in P^ - probability 
~ p° 

jor any 9 — > t'o- 
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SEMIPARAMETRIC BOOTSTRAP CONSISTENCY 15 

Verifications of Conditions SB1-SB2 will be discussed in Section [3l Two 
general Theorems are given in Section [5] to aid verification of Condition SB3. 
To rigorously state the boots trap consistency result, we need the notion 



of conditional weak convergence [16|]. Let BLi{M) be a collection of Lipschitz 
continuous functions /i : B M bounded in absolute value by 1 and having 
Lipschitz constant 1, i.e. < 1 and \h{x) — h{y)\ < \\x — y\\ for all 

x,y £ M. We say X* converges weakly to X conditional on the data Xn , 
denoted as X* =^ X, if 

(23) sup \E.\:^^h{X:)-Exh{X)\=o"p^{l), 

heBLi{M) 

where E.\x„ denotes the conditional expectation given the data Xn, provided 
h{X*) is asymptotically measurable unconditionally for all h G BLi{M). 



More discussions of conditional weak convergence can be found in 
The next result is a bootstrap version of Theorem [TJ 

Theorem 2. Suppose that 9 and 9* satisfy and ^8\). respectively. 

^ Px ^ ^xw 

Also assume that 9 — > ^.i^d 9* — > ^o- Assume that Conditions I, S1-S3, 

SB1-SB3 and W1-W5 hold. We have that 

(24) \\P-9o\\ = 0°p^in~y') 
in -Probability. Furthermore, 

(25) V^i9* -9) = -A-^Glmo + 0"p^ (n'/^-^^) 
in P^ -probability. Thus, 

(26) (V^/c)(r-^)^iV(0,S), 

where c is given in W5, whose value depends on the used sampling scheme, 
and S = A'^B{A-^y with A and B given in Condition I. 
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^ p° 

The assumption 9* — > On can be established by adapting the Argmax 



Theorem, Corollary 3.2.3, in 



3j| . Briefly, we need two conditions for accom- 



plishing this. The first one is the "well separated" condition (llSp . Another 



one IS 



(27) sup |P;m(6', r])-Pxm{e, ?/) | ^ 0. 



By the Multiplier Glivenko-Cantelli Theorem, i.e. Lemma 3.6.16 in [3J], and 
(|A.ip in the Appendix, we know that ^ holds if {m(6', r]) : 6 e @,r] e H} 
is shown to be P-Donsker. 

Note that ^ and ^ of Theorem [2] are in parallel to ^ and (HH) of 
Theorem [TJ In particular, the asymptotic linear expansion (j25p is a second 
order one in the sense that the remainder term is of more refined order than 
Op^, (1) whose rate of convergence depends on how accurately the nuisance 
function i] can be estimated. Thus, it might be reasonable to conjecture that 
more accurate bootstrap inferences can be drawn from the semiparametric 
models with faster convergent i]. 

Let Pw\Xn denote the conditional distribution given the observed data 
Xn- Note that Theorem [2] implies that 

(28) sup PwixMV^/c){0* -9) < x) - P{N{0,J:) < x) =o?>^(l) 

by setting h{-) = 1{- < x}, where " < " is taken componentwise, in ([23]). 



Theorem [T] together with Lemma 2.11 in 



351 ] implies that 



(29) sup PxiVn(e-eo) <x)- P{N{0,^) <x) = o(l). 



Combining ()28p and (1290 . we obtain the following bootstrap consistency 
result. 
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Corollary 1. Under the conditions in Theorem{^ we have that 

(30) sup Pw\xMV^/^W -0)<x)- PxiV^iO- Oo) < x) ^ 



as n ^ oo. 



Corollary [T] says that the bootstrap distribution of (^/n/c){9* — 6) asymp- 
totically imitates the unconditional distribution of y/n{9 — 9o). 

Remark 3. For any consistent S* — > T, and S — > T,, we have 
(31) (v/S/c)(E*)-"=(r-9)^W(0,/), 



Pi 



(32) sup 

X 

by Theorem Slutsky 's Theorem and the arguments in proving Corollary [71 
An app ropriate candidate for the consistent S* is the block jackknife proposed 



in IMl- 



2.4. Bootstrap Confidence Sets. In this subsection, we show that the dis- 
tribution consistency of the bootstrap estimator 9* proven in Corollary [T] im- 
plies the consistency of a variety of bootstrap confidence sets, i.e. percentile, 
hybrid and t types. 

A lower a-th quantile of bootstrap distribution is any quantity t:*^, £ 
satisfying t*^ = inf{e : Py/\Xn{^* < e) ^ Q;}, where e is an infimum 
over the given set only if there does not exist a ei < e in M"^ such that 
Pw\Xni^* — ^i) — CK- Because of the assumed smoothness of the criterion 
function m{9, rj) in our setting, we can, without loss of generality, assume 
Pw\x,^(^* < 'Tna) = We can also define k*„ = {\/n/c){T*^ - 9) so that 
-fV|A'„((\/^/c)(^* — 9) < K*Q,) = a. Note that r*^, and k*^, are not unique 
since we assume ^ is a vector. 
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Recall that, for any x G M*^, 

Px{V^{0-9o)<x) ^{x), 
Pw\x^{V^lcW -9)<x) % ^{x), 

where ^(x) = P(iV(0, S) < x), by ([H]) an d ((28l) . By mvoking the quantile 
convergence Theorem, i.e. Lemma 21.1 in 



35l ]. we know k*^ ^"-"^(a). 
Considering the Slutsky's Theorem which imphes v^l^— ^o) — ^J^(q,/2) weakly 
converges to N{0, S) - ^'-i(a/2) unconditionally, we further have 

Pxw Oo<e- 'S^] = (^(0 _ > ^* 



77, 



Pxw{N{0,^)>^~\a/2)) =l-a/2. 

The above arguments prove the consistency of the hybrid-type bootstrap 
confidence set, i.e. ()34p . and can also be applied to the percentile-type boot- 
strap confidence set, i.e. (j33p . More rigorous proof can be found in Lemma 



23.3 of 



35l |. The following Theorem [3] summarizes the above discussions. 



Theorem 3. Under the conditions in Theorem\^ we have 

(33) PxH^lg+ '""^"^'^~^ <go<^+ ^"^'""^^'^~ I — 

(34) Pxw{e-^^^^^<e,<9-^^^\ 1-a, 



as n ^ oo. 



n yjn 



It is well known that the above bootstrap confidence sets can be obtained 
easily through routine bootstrap sampling. 

Investigating the consistency of the bootstrap variance estimator is also 
of great interest. However, the usual sufficient condition for moment consis- 
tency, i.e. uniform integrability condition, becomes very hard to verify due to 

imsart-aos ver. 2006/01/04 file: bconsistency-JH_v6.tex date: June 6, 2009 



SEMIPARAMETRIC BOOTSTRAP CONSISTENCY 19 

the existence of an infinite dimensional parameter r]. An alternative resam- 
pling method to obtain the bootstrap variance estimator in semiparametric 
models is the block jackknife approach, which was proposed and theoreti- 
cally justified in 2J]. We do not pursue this topic further in this paper. 



Remark 4. Provided consistent variance estimators S* and T, are 
available, we can prove that the following t-type bootstrap confidence set is 



also consistent by applying Lemma 23.3 in 



J to (D^ and (31\): 



Vl/2, ,* Vl/2, ,* 

'n Jn 



as 



oo, where lv^^ satisfies Pw\xAiVn/c)i9* - < w^^) = a. 



3. Verifications of Conditions S1-S2 and SB1-SB2. In this sec- 
tion, we discuss how to verify Conditions S1-S2 and SB1-SB2 in Subsec- 
tion 13.11 and 13.21 respectively. Two general theorems are given in Section H] 
to assist verifying the remaining convergence rate conditions S3 and SB3. 

3.1. Verifications of Conditions S1-S2. The continuity modulus condi- 
tion Q in SI can be checked via one of the following two approaches. 
The first approach is to show the boundedness of £'^||G„||5„ by using 



Lemma 3.4.2 in 



34i |. The second approach is to calculate the bracketing 



36| if L2-norm is used on 



entropy number of 5„ and apply Lemma 5.13 in 
the nuisance parameter. As for (jlOp . we can verify it easily if we can show 
that the class of functions {{d / d0)m{9 , r]) : {9,r]) G C„} is P-Donsker. 

Next, we discuss how to verify the smoothness condition S2. We first write 
Pxifh{9, Tj) — mo) as the sum of Pxifh{9, rj) — fh{9Q,r])) and Px{m{9Q, rj) — 
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mo). We apply the Taylor expansion to Px{m{6,r]) — m{9Q,7])) to obtain 

Px{m{9,r]) - m{0o,7])) 
= Px {mn{Oo,7]) - m2i{9o,rj)[H\eo,v)]} {0 - Oo) + 0{\\9 - Oof) 

= Ai6 - Oo) + oi\\e - Oof) + {e- eo)o{\\v - %||), 

where A is defined in Condition I, the first and second equality follows from 
the Taylor expansion of 9 Px'fh{9,r]) around 9q and 

V^Px {mn{9o,r]) - m2ii9o,v)[HH9o,v)]} 

around r/o, respectively. By applying the second order Taylor expansion to 
rj I— > Pxfh{9Q, rf) around 770 and considering (j4]), we can show that P{rh{9o, rj) — 
mo) = 0{\\r] — r/o|p). In summary, Condition S2 usually holds in models 
where the map rj ih{9Q,r}) is smooth in the sense that the Frechet deriva- 
tive 7] Px{{d/d9)m{0Q,r])) around rjo and the second order Frechet 
derivative of 77 1— > Px'm{9o,r]) around rjo are bounded as discussed above. 

3.2. Verifications of Conditions SB1-SB2. We can verify Condition SBl 
by showing either Sn{x) is uniformly bounded, i.e., limsup„_^oo 'S'n(2^) < 
M < 00 for every x ^ X, more generally, limsup„^oo E{Sn{Xi)^^^} < 00 
for some 6 > 0. That the moment condition implies Condition SBl follows 
from the Chebyshev's inequality. In our examples in Section [5l the uniformly 
boundedness condition is usually satisfied. Hence, we focus on how to show 
Sn(x) is uniformly bounded here. By the Taylor expansion in a Banach space 
40|, we can write rh{9o, 77) — mo = D~[r]—rio], where rj lies on the line segment 
between r] and rjo, and -^^[/i] is the Frechet derivative of 77 ^ m.{9o,rj) at 
^ along the direction h. Since we require II77 — 770 II < (^n ^ 0, the bounded 
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Prechet derivative at r/o will imply that Sn{x) is uniformly bounded. The 
method in verifying (jlOp of Condition SI can be applied to check Condition 
SB2; see the discussion in the previous subsection. 

4. Convergence Rates of Bootstrap Estimate of Functional Pa- 
rameter. In this section, we present two general theorems for calculating 
the convergence rate of the bootstrap estimate of the functional parameter. 
These results can be applied to verify Condition SB3. The convergence rate 
condition S3 can also be verified based on these theorems by assuming the 
weights Wni = 1 for i = 1^- Note that both theorems extend general 
results on M-estimators 



271, 



34l | to bootstrap M-estimators and are also of 



independent interest. 

4.1. Root-n rate. In the first theorem we consider a collection of mea- 
surable objective functions x ^ k{9,ri)[g\{x) indexed by the parameter 
{0,r]) & Q xTC and an arbitrary index set g £ G. For example, k{6, 'r])[g] can 
be the score function for rj given any fixed indexed hy g £ G. Define 

U*ie,r])[g] = Flki6,r^)[g], 
Un{e,vM = Fnki9,r,)[g], 
U{e,i])[g] = Pxk{e,ri)[g]. 

We assume that the maps g i-^ U*{9,r])[g], g Un{0,'rj)[g] and g 
U {6, 'r])[g] are uniformly bounded, so that U*, Un and U are viewed as maps 
from the parameter set Q x Ti. into i°°{G). The following conditions are 
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assumed in Theorem |4] below: 

(35) {k{e,r,)[g\ : \\e - 9^ + \\r, - r]^ < 6, g e G} £ M{Px) n L^iPx) 
and is P-Donsker for some 6 > 0, 

(36) snpPx{kie,ii)[g]-k{eo,vo)[g]V ^0 as \\e - OqW + \\v - Vo\\ ^ 0- 

Let 

f A:(g,77)[g]-A:(go,r?o)[g] 
L 1 + y/n\\e - fc^oll + y/n\\ri - r/o|| 

and Dn{X) be the envelop function of the class of functions P„. For any 
sequence 5„ — > 0, we assume that Dn{X) satisfies 

(37) lim lim sup sup tVl(L»„(Xi) > t) = 0. 

A— >oo n— >oo t>A 

Now we consider the convergence rate of rf^ satisfying: 



(38) K{0.^~)[9]=O"p^^{n-'/^) 
~ p° 

for any 9 — > 9 ranging over G. In the below Theorem we will 

show that rjt has the root-n convergence rate under the conditions (|35|) - (|37|) . 

Theorem 4. Suppose that U : Q x H ^ £°°{G) is Frechet differ- 
entiable at (OotVo) with bounded derivative U : 'U.'^ x linTi ^ £°°{G) such 
that the map U{0, •) : linTC h-> i°°(G) is invertible with an inverse that is 
continuous on its range. Furthermore, assume that l35\) -i'3l) hold, and that 
U {Oo,r]o) = 0, then 

(39) ll?7~-%ll=Op^(ll^-eo||Vn-V2) 

po po 

in -probability, given that 6 and fj^ 770- 
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4.2. Slower than root-n rate. We next present another Theorem which 
yields slower than -y/n convergence rate for the bootstrap M-estimate of the 
functional parameter. This result is so general that it can be applied to 
the sieve estimate of the nuisance parameter The essence of the sieve 



method is that a sequence of increasing spaces (sieves), i.e. TLn, is employed 
to approximate the large parameter space, e.g. 7i, so that asymptotically, 
the closure of the limiting space contains the original parameter space. In 
other words, for any rj £ 7i, there exists a 7r„r/ G Tin such that H?] — 7r„?7|| — > 
as n — > oo. 

Now we consider the M-estimate fj^ £ Tin satisfying 

(40) Kv{0, ?te) > Kv{e, 7]n) for any 61 G G and some ??„ e Hn, 

where x ^ v{6,r]){x) is a measurable objective function. Let " ^ " and 
" ^ " denote greater than or smaller than, up to an universal constant. We 
assume the following conditions hold for every 5 > 0: 

(41) Ex{v{e,7])-v{e,7]n)) ^ -d\r],rjn) + \\e-eof, 



(42) E% sup \Gn{v{e,r,)-v{9,rin))\ <MS), 

ee0,v(^Hn,\\0~0o\\<iAv,Vn)<s 



(43) sup \GUv{0,v)-v{e,r]n))\ <rn{S)- 

e&e,riG'H,i,\\e-eo\\<S,d{ri,rin)<5 

Here d^(r?, ??n) may be thought of as the square of a distance, i.e. ||?7 — r/nlPj 
but our theorem is also true for any arbitrary function 77 d?[r],r]n)- 

Theorem 5. Suppose that Conditions ^JJ^-^J^ hold. We assume 
(4^ ) ((4^)) is valid for functions ipn (V'n) such that 5 ^ ^„((5)/5" {5 ^ 
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'4'ni^)/^°') ^-^ decreasing for some < a <2. Then, for every {0,rf-) satisfy- 
ing P{9 ^ Q,rj~ Tin) — > 1, we have 

6 

d{rj*~,rjn)<0'i>^X^ny\\e-9o\\) 

in Px -probability, for any sequence of positive numbers 5n satisfying both 
tpn{Sn) < \fnb\ and tpni^n) < V^^l for every n. 

In application of Theorem[5l the parameter rjn is taken to be some element 
in Tin that is very close to rjQ. When Tin = Ti, a natural choice for rjn is rjQ 
and we can directly use Theorem [5] to derive the convergence rate d{fj~,rjQ) 
as shown in the examples of Section [3 In general, rjn may be taken as the 
maximizer of the mapping rj i— > Pxv{9o, rf) over Tin, the projection of rjQ onto 
Tin- Then we need to consider the approximation rate of the sieve space Tin 
to Ti, i.e. d{r]n,r]o), since d{fj^, tjq) < d(j}~^ r]n)+d{r]n,r]o). The approximation 
rate d{rjn,r]Q) depends on the choices of sieves and is usually derived in the 
mathematical literature. Here, we skip further discussion and refer readers 
to [4|. 

Now we discuss verification of the nontrivial conditions (|4ip - (|43|) . The 
smoothness condition for v{6,r]), i.e. (j4ip . is implied by 

(44) Ex{v{d,r,)-v{eo,r,n)) < -d\r,,r,n) - \\d - Oof , 

(45) Ex{v{e,r]n)-v{9o,Vn)) ^ "11^ -^of- 

The two conditions depict the quadratic behaviors of the criterion functions 
{6, rj) I— > Exv{0, rj) and 6 Exv{6, r]n) around the maximum point (^O) Vn) 
and respectively. We next present one useful lemma for verifying the 
continuity modulus of empirical processes and its bootstrapped version, i.e. 
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42]) and (HSl). Denote 



(46) Vs = {x^ [v{e,iM^) - vi9,Vn){x)] : d(r?,r/„) < S, \\9 - OqW < 5}. 
and define the bracketing entropy integral of as 

(47) K{5,Vs,L2iPx)) = 1^ ^l + logN[]{e,Vs,L2iPx))de, 

where log A'^[] (6, A, d) is the 5-bracketing entropy number for the class A 
under the distance measure d. 

Lemma 1. Suppose that the functions {x, 9, rj) i— > vg^riix) are uniformly 
hounded for {9,r]) ranging over some neighborhood of {9Q,rjn) and that 

(48) Ex{ve,^ - vg^^S ~ d^i'H.'nn) + \\9 - ^of • 

Then Condition (JW satisfied for any functions ipn such that 

(49) MS) > mVsMPx)) (l + ^^^^^^1^^) • 

Let Vn{X) he the envelop function of the class Vs„- If we further assume 
that, for each sequence 5n 0, the envelop functions Vn satisfies 

(50) lim lim sup supt'^P^{VniXi) > t) = 0, 

A-^oo n— >oo t>X 

then Condition 1^3] ) is satisfied for any functions such that 

(51) MS) > Ki6,Vs,L2iPx)) (l + ^^^^^1^^) • 

Remark 5. Note that the inequalities t/^niS) ~ \^6'^ and t/j^iS) ~ ^fub"^ 
are equivalent with K{6,Vs, L2{Px)) ~ \/n6'^ when we let ipn o,nd ip* 
equal to right hand side of (J^ and (f57]). respectively. Consequently, the 
convergence rate ofrj- calculated in Theorem\^ i.e. 6n, is determined hy the 
bracketing entropy integral of Vs„ • 
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Remark 6. The assumptions of Lemma Ul (ire relaxahle to great ex- 
tent. For example, we can drop the uniform hounded condition on the class 
of functions v{9,r]) by using the "Bernstein norm", i.e. ||/||p,_b = (2^(61-^1 — 
1 - instead of the L2 -norm. In some cases, the bracketing entropy 

integral diverges at zero. Then we can replace by the integral: 

Ic5^a5/3 



K{6,Vs,L2{Px))= Jl + logN[]{e,V5,L2{Px))de, 



for some small positive constant c, see Lemma 3.4-3 and page 326 in \3^ ]. 



5. Examples. In this section, we apply the main results in Section[2]to 
justify the bootstrap validity of drawing semiparametric inferences in three 
examples of semiparametric models. In the first two examples, we use the 
log-likelihood as the criterion function, while in the last example, the least 
squares criterion is used. The M-estimate of the nuisance functional param- 
eters have different convergence rates in these examples. This section also 
serves the purpose of illustration on verification of the technical conditions 
used in the general results. 

5.1. Cox Regression Model with Right Censored Data. In the Cox re- 
gression model, the hazard function of the survival time T of a subject with 
covariate Z is modelled as: 

(52)A(t|z) = hm^ ^P{t <T <t + A\T >t,Z = z)= X{t) exp{e'z), 

where A is an unspecified baseline hazard function and is a regression 
vector. In this model, we are usually interested in 9 while treating the cu- 
mulative hazard function r/(y) = X(t)dt as the nuisance parameter. The 
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MLE for is proven to be semiparametric efficient and widely used in ap- 
plications. Here we consider bootstrapping 6, which corresponds to treating 
log-likelihood as the criterion function m{6,r]) in our general formulation. 

With right censoring of survival time, the data observed is X = (Y, 6, Z), 
where Y = T A C, C is a censoring time, 5 = I{T < C}, and Z is a 
regression covariate belonging to a compact set Z, C M'^. We assume that C 
is independent of T given Z. The log-likelihood is obtained as 

(53) m{9,r]) = 69' z — exp{9'z)r]{y) + 6 log ri{y}, 

where r]{y} = rj(y) — r]{y—) is a point mass that denotes the jump of r] 
at point y. The parameter space 7i is restricted to a set of nondecreasing 
cadlag functions on the interval [0, r] with r/(r) < M for some constant M. 
By some algebra, we have 

fh{0,ri){x) = mi{9,r]) - m2{9,i])[H\9,i])] 
= [6z — z exp{9' z)rj{y)] 

- 6H^9,ri){y)-exp{9'z) j ' H^9,r]){u)dr]{u) 

where 







hU9 ,)iy) = Ee,r,ZeMO'Z)l{Y>y} 

Conditions I, S1-S3 in guaranteeing the asymptotic normality of 9 have 
been verified in 6|]. In particular, the convergence rate of the estimated 
nuisance parameter is established in theorem 3.1 of j^^], i.e. 

(54) 11%^ - %l|oo = Op^{n~^ + \\9n - 9o\\), 

where || • ||oo denotes the supreme norm. We next verify the bootstrap con- 
sistency conditions, i.e., SB1-SB3. Condition SBl trivially holds since it is 
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easy to show that rj fa{9Q,rj) has bounded Prechet derivative around 
r/o- The P-Donsker condition SB2 has been verified when verifying (jlOp 
in condition SI. In the end we will verify the bootstrap convergence rate 
condition \\rf~ — r/o||oo = 0°p^^{\\6 — 6q\\ V 77,"^/^) via Theorem [H Since ifg 
maximizes f*^m[0,i]) for fixed 6, we set k{9,r])[g\ = in2{9 ,r])[g\ and have 
U*{e,rjl)[g] = ¥lm2{e,r]l)[g] = 0. The invertibility of W{Q,-), the condi- 
tions ()35p and (j36p have been verified in when they showed (|54p . Now 
we only need to consider the condition (j37p : for n so large that 5n < R 



The last inequality follows from the assumption that G is a class of func- 
tions of bounded total variation and the inequality that g{u)dr][u) < 
r]{T)\\g\\Bv 1 where ||5||_By is the total variation of the function g. Thus, the 
condition (I37p holds trivially. 

5.2. Cox Regression Model with Current Status Data. We next consider 
the current status data when each subject is observed at a single examination 
time C to determine if an event has occurred. The event time T cannot be 
known exactly. Then the observed data are n i.i.d. realizations of X = 
(C, 5, Z) e R+ X {0, 1} X Z, where <5 = I{T < C} and Z is a vector of 
covariates. The corresponding criterion function, i.e. the log-likelihood, is 
derived as 



(55) m{6, T]) = (51og[l — exp(— r/(c) exp{9'z))] — {1 — 6) exp{6' z)r]{c). 
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< some constant. 
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We make the following assumptions throughout the rest of this subsection: 
(i) T and C are independent given Z; (ii) the covariance of Z — E{Z\C) is 
positive definite, which guarantees the efficient information to be positive 
definite; (iii) C possesses a Lebesgue density which is continuous and posi- 
tive on its support [a, r] , for which the true nuisance parameter 770 satisfies 
r/o(c— ) > and %(''") < M < 00, and this density is continuously differen- 
tiable on [a, rl with derivative bounded above and bounded below by zero. 

~ Q 

The form of m{9,rj) can be found in [7|] as follows 

fh{6,r]) = mi{6,r]) — 1712(0, ri)[H'^ {6, T])] 
= izr,{c)-HHe,r,)ic))Qix;e,v) 

where 

Q(x; e, ??) = e''^ U , g,/, , - (1 - 5) 

L exp^e*^ 'r](c)) — 1 

and the form of {9,r]){c) is given in (4) of Q]- 

Conditions I and S1-S3 are verified in [3]. Conditions SBl and SB2 can 

be checked similarly as in the previous example. Note that the convergence 

rate for the nuisance parameter becomes slower, i.e. 

(56) 11% - ??o||2 = Op^iWOn - OoW+n-'/^), 

where || • II2 denotes the regular L2-norm, due to the less information provided 



by the current status data, as shown in [27[. By Theorem[5]we can show that 
the same convergence rate, i.e. n~^/^, also holds for the bootstrap estimate 
'or 77, i.e. rf^. The assumptions ()4ip and (j42p in Theorem [5] are verified in 
27I 1 when showing (j56p . As for the last assumption (j43p . we apply Lemma [TJ 
We show that Condition (jSOp on the envelop function Vn{x) holds: for n so 
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large that 5n< R 

Vn{x) = suY){\m{9,7]) - m{9,7]Q)\ \\\ri - riQ\\2 <5n,\\0 - 9q\\ <5n] 
< 2sup{|m(0,r?)| : Wn - < RA\e - 9^ < R} 



5.3. Partly Linear Models. In this example, a continuous outcome vari- 
able y, depending on the covariates {W, Z) G [0, 1]^, is modelled as 



where is independent of (W, Z) and / is an unknown smooth function 



belonging to H = {f : [0,1] ^ [0,1], j^{f^^\u)Ydu < M] for a fixed 



< A/ < cxo. In addition we assume E{Var{W\Z)) is positive definite and 
E{f{Z)} = 0. We want to estimate {6,f) using the least square criterion: 



Note that the above model would be more fiexible if we did not require 
knowledge of M. A sieve estimator could be obtained if we replaced M with 
a sequence M„ — ^ oo. The theory we develop in this paper will be applicable 
in this setting, but, in order to maintain clarity of exposition, we have elected 
not to pursue this more complicated situation here. An alternative approach 
is to use penalization, the study of which is beyond the scope of the present 
paper. 

Simple calculations give 



< some constant. 



Y = 9W + f{Z)+C 



(57) 



m{9J) = -{y-ew-f{z)f. 



m{9, rj){x) 



mi{e,r])-m2{e,r])[H\e,r])] 
2{y-9w-f{z)){w-HHe,v){z)), 
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where 

rrUn _ Eg^,{WiY-9W-fiZ)r\Z = z) 

M[U,r^)[z)- E^^^^^Y-eW-f{Z))^\Z = z) ' 

The finite variance condition I follows from E[W{W - H^Oq, r]o){Z)}] > 0. 
The distribution of ^ is assumed to have finite second moment and sat- 
isfy ([6]), e.g. ^ ~ A^(0, 1). Conditions S1-S3 and SB2 can be verified us- 
ing similar arguments in Example 3 of Q], in particular, — /olb = 
Op^{\\9 - OqW V n-''/(2fc+i)) ^ gj^gy ^Yiow that the Prechet 

derivative of r/ 1-^ ffi{9Q, rj) is bounded around i]q, and thus the tail condition 
SBl holds. To prove ||/| - /o||2 = 0°p^^{\\9 - OqW V n-'=/(2fc+i)) via Theo- 
rem [5l we proceed as in the previous example, checking the assumption ()50p 
using similar arguments, i.e. Vn{x) is uniformly bounded. 

6. Proof of Main Results. 

6.1. Proof of Theorem{l\(Semiparametric M-estimate Theorem). By Con- 
ditions SI, S3 and the consistency of 0, we have 

Gn{m - mo) = 0"pJ\\9- 9o\\ V ||r? - r?o||) = 0?>^-(||^- ^o|| V n"^). 

Considering ([5]), we can further simplify the above equation to 

(58) V^Pxim - mo) = -V^F^mo + 0"p^{\\9 - 9o\\ V n"^). 

By Conditions S2-S3 we can show that the left hand side of (I58p equals 

V^Px{mn{9o,rio) " "i2i(^o, %)[^o^])(^- 9o) + 0°p^ {V^i\\9 - ^of Vn^/^-ST) 
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Thus (158]) becomes 

V^Px{mu{Oo,Vo) - m2i{9o,Vo)[Hl])(e - Oo) 

(59) = -V^Fnmo + 0"p^{\\d-eo\\Vn-^)+0"p^{V^\\d-eofVn^/^-^^) 
= -V^Fr^mo + 0?>^(ni/2-27 V \\9-0o\\ V M\0 - Oof) 

(60) = o''pji\/V^\\e-eof), 

where the second equahty follows from the range of 7 and the third equality 
holds by applying CLT to mo- Due to the consistency of 6 and condition I, 
we can conclude that — 0o|| = Op^{n~^/^) based on (f60]) . thus ||?? — 7?o|| = 
0'p^{n~^). Plugging the convergence rates for 9 and rj back to (j59p . we 
complete the proof for (113p . The standard CLT and ([6]) implies ()14p . 

6.2. Proof of Theorem (Bootstrap Consistency Theorem). To prove 
Theorem [2l we need the following lemma whose proof is given in the Ap- 
pendix. 

Lemma 2. Under the assumptions in Theorem\^ we have 

(61) G; (m(0,7?) - m{Oo,m)) = 0"p^{\\e - O^W V \\r, - r,o\\) 
in -Probability for (9, rj) (z Cn- 

We shall use repeatedly Lemma lA.ll in the Appendix, which concerns 
about the transition of stochastic orders among different probability spaces. 

We first prove Recall that G„ = ^(P„-Px) and G; = V^(P;-P„). 
Define m* as ifi{9*,r]*). By some algebra, we have 

G*mo + GnTO-o + ^/nPx{fh* - fho) 
= G* (mo — m*) + G„(mo — m*) + >/nP*m*, 
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since Pxfn-o = 0. Thus we have the below inequahty: 

\\^Px{fh* - mo)|| < IIG^moll + ||G„mo|| + ||G;(mo - fh*)\\ 
(62) +||G„(^o_a*)|| + ||V^P;m*|| 

< Li + L2 + L3 + L4 + L5. 

Based on Theorem 2.2 in 



2g], we have Li = 0°p^^.{^) in PJ-probabiHty. 
The CLT imphes L2 = 0°p^{l). We next consider L3 and L4. By Condition 
SB3, we can show that \\r]* — rj^W = Op^(l) in PJ-probabihty since 9* is 
assumed to be consistent, i.e. 116** — ^oll = Op^_(l) in PJ-probabihty, by (jA.ip 
and (jA.SP in Lemma lA.ll Then, we have L3 = o°p^Xl) in P^'P^obabihty 
based on Lemma [2] and (jA.SP in Lemma lA.li Next, we obtain that L4 = 
Op^(l) in P^-probabihty based on Condition SI and (IA.3P in Lemma lA.li 
Finally, L5 = based on ([18]) . In summary, ([62]) can be rewritten as: 

(63) W^p^i^fk* -m,)\\<0°p^X\) + 0°p^{l) 
in P^-probability. 

Let an = 11^* — 6*0 II . Combining (fTT|) with ([55]) and noticing ([^ . we have 

(64) V^Pa„|| < 0?>^(1) + O^^(l) + O?,^. (V^a^ V n-^"'+^/^) 

in P^-probability. By considering the consistency of 0* and Condition I, we 
complete the proof of (p^ based on 
We next prove ([25]). Write 



Ii = -G;(m*-mo) = V^(P:-IPn)(mo-m*), 

-^2 = G„(m - mo) = \/n(P„ - Px)(m - mo), 

4 = -G„(m* - m-o) = V^(Pn - Px){fho - m*), 

I4 = ^p;m* - V^P„7n. 
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By some algebra, we know that ^/uPxiiri* — in) + G*mo = J2'j=i ^j- 

By the definition ([20]) . we can show that An x Bn = Op^(l) in P^"- 
probabihty if An and Bn are both of the order Op^(l) in P^-probabihty. 
Then the root-n consistency of 9* proven in (|24p together with SB3 imphes 

(65) \\v*-r]o\\'y\\0*-0o\\ = O*p^in-^) 

in P^'P^'obabiUty. Thus, by Lemma [21 we know Ii = 0°p^Xn~^) in PJ- 
probabihty. Note that ([9|) and (jlOp of Condition SI imply 

(66) Gn(m(0,??) - mo) = 0"p^{\\e - O^W V ||7? - 7?o||) 

for (0, T]) in the neighborhood of (^O) %)• Considering ([66]). Condition S3 and 
Theorem m we have h = 0°p^{n-^). By ([MD, ([MI) and (TO]) , we know the 
order of Is is 0°p^{n-^) in P|-probability. /4 = by (0) and (USD. 
Therefore, we have established: 

(67) y/^Px (m* -m) = -G^mo + Op^ {n~^') + 0"p^^ (n"^) 

in P^-probability. To analyze the left hand side of (|67|) . we rewrite it as 
^JnPxirn* — fho) — \/nPx{m — fho). Applying Condition S2 to both compo- 
nents, we obtain 

{eo,Vo) -m2i{eo,Vo)[Hl]W -6) 
= -Glmo + 0|>,, (n-^) + O?,^, (n"^) + 0°p^ (n V^-^t) + qo^ (^1/2-2,) 
= -G:mo + 0|,^(ni/2-27) + oo^(„i/2-2,) 

in P^-probability, by considering Condition S3, SB3 and the range of 7. By 
considering Conditions I and ()A.2p . we complete the proof of (|25p . The proof 



of (1261) follows from Theorem 2.2 in 



29|. 



APPENDIX 
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A.l. Measurability and Stochastic Orders. 

Measurahility Condition M{P) We say that a class of random functions 
JT G M{P) if JF is nearly linearly deviation measurable for P and that both 
T"^ and J-'"^ are nearly linearly supremum measurable for P. Here J-'^ and J-'"^ 
denote the classes of squared functions and squared differences of functions 
from respectively. It is known that if T is countable, or if \f'n}^-^ are 



id], then 



stochastically separable in J^, or if !F is image admissible Suslin 
T G M{P). More precise descriptions can be found in Page 853-854 of []J 

The following lemma is very important since it accurately describes the 
transition of stochastic orders among different probability spaces. We im- 
plicitly assume the random quantities in Lemma [A. 1 1 posses enough measur- 
ability so that the usual Fubini theorem can be used freely. 

Lemma A.l. Suppose that 

Qn = o°p^{l) in Px -Probability, 

= 0°p^{l) in P"^- Probability. 

We have 

(A.l) An = o°p^^Xl) ^ An = o°p^.{l) in P^-Probability, 

(A.2) Bn = 0"p^^{l) ^ Bn = 0''p^^,.{l) in P^-Probability, 

(A.3) Cn = Qn X O^p^il) =^ Cn = o?,^,(l) m P^ -Probability , 

(A.4) Dn = RnX Op^ (1) =^ Dn = 0°p^ (1) in P^ -Probability, 

(A.5) En = QnXRn =^ En = o°p^^,{l) in Px - Probability . 
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Proof: To verify (jA.lh . we have for every e, > 0, 

P°x{Pw\x{\M>e)>^} < lE°xPw\xi\An\>e) 
(A.6) < ^E°xE^^,,l{\An\ > e} 

based on the Markov's inequahty. Based on Lemma 6.5 and 6.14 in [l^, we 
have E°^E^^^l{\Ar,\ > e} < > e} = P^wHA^l > e), and thus 

(A.7) PI {P^\xi\^n\ >e)>u}< ipVd^nl > e). 

/,From (lA.Tp . we can conclude that if An = Op^^(l), then An = Op^(l) 
in P^-probabiUty. Another direction of ()A.ip foUows from the following in- 
equalities: for any e,7] > 0, 

P°xwi\An\>e) = E'k{P^^xi\An\>e)} 
= ES,{P^^xi\An\ > e)l{P^|^(|A„| > 6) > r?}} 
+EUP^lxi\An\ > e)l{P^|x(l^n| > e) < r?}} 

(A.8) < P^{P^{\An\>e)>v} + 7i. 

Note that the first term in (jA.Sp can be made arbitrarily small by the as- 
sumption that An = Op^(l) in -Probability. Since rj can be chosen ar- 
bitrarily small, we can show lim„^oo Px"vk(I^™I — = for any e > 0. 
This completes the proof of (jA.ip . (|A.2p can be shown similarly by using 
the inequalities (|A.6p and (jA.Sp . 
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As for (IA.3p . we establish the fohowing inequahties: 
P^{Pw\xi\QnXO"pJl)\>e)>u} 

< P"x{Pw\xi\Qn\>^/\0''pJl)\)>>^} 

< Px{Pw\xi\Qn\ > e/M) + > M) > z.} 

< P°x{P^ixi\Qn\ > e/M) > u/2} + P^{P^^j,i\0"pJl)\ >M)> u/2} 

< PUPw\xi\Qn\ > e/M) > u/2} + lp^{\0"pjl)\ > M) 

for any e, z^, M > 0. Since M can be chosen arbitrarily large, we can show 
()A.3P by considering the definition of 0°p^{l). The proof of ()A.4p is similar 
by using the above set of inequalities. The proof of (IA.3|) can be carried over 
to that of (jA.Sp . Similarly, we establish the below inequalities: 

Px {Pw\x{\Qn X Rn\ > e) > r?} 
< Px{Pw\xi\Qn\ > e/M) > r?/2} + P^{P^^xi\Rn\ > M) > 7?/2} 

for any €,7],M > 0. Then by selecting sufficiently large M, we can show that 
Px {Pw\x(\Qri X Rn\ > e) > 7/| ^ as n ^ cx) for any e, > 0. □ 

A. 2. Two useful inequalities. Here we give two key technical tools. Mul- 
tiplier Inequality and Hoffmann- J orgensen Inequality^ used in proving Lem- 
mas [U and [2j 

Multiplier Inequality (Lemma 4-1 of 131]) 



Let Wn = {Wni, ■ ■ ■ jWnn)' be non-negative exchangeable random vari- 
ables on (W, 0, Pw) such that for every n 

Rn= \l Pw {Wnl > U)du < OO. 

Let Zni, i = 1,2,..., re, be i.i.d. random elements in (X°°,A°°,Px) with 
values in £°°{J^n), and write || • ||„ = supj^j^-^ It is assumed that 
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Zni^s are independent of Wn- Then for any no such that 1 < no < oo and 
any n > no, the following inequality holds: 



xw 



1 



i=l 



< noEx\\Zni\ 



£'vi/(maxi<i<„ Wn 



n 



(A.9) 



no<«<n 





i 




jo+1 



IMD 



Hoffmann- Jorgensen Inequality for Moments (Prop. A. 1.5 in 
Let 1 < p < oo and suppose that Vi, . . . ,Vn are independent stochastic 

processes with mean zero indexed by an arbitrary index set T. Then there 

exist constants Kp and < Vp < 1 such that 



4 = 1 



<i^p i?" max \\Vkr + [G-\vp)r 

l<k<n 



where || • || denotes the supremum of a stochastic process {Yt,t G T}, i.e. 
supj I If I and 



G"\v) = ini \u : P" i\\Y,Vi\\<v] > 



k \ i=l /J 

A. 3. Proof of LemmalM We first write G* (rn(^, rj) — mo) as the sum of 
Gn{Th{9,T]) — fh{6o,ri)) and G* (rn(0Oi ^) — ""t-o)- By the Taylor expansion, the 
first term becomes {9 — 9{))'Qn[d/d6)fh{0, rf), where 9 is in between 9 and 9q. 



29( 1 . we know the first term is 



By the assumption SB2 and Theorem 2.2 in 
of the order 0°p^{\\9 — 9q\\) in P^-probability. We next consider the second 
term. Let 



(A.IO) 



A. 



sup 



\{m{9o,r]) - mo) 



11^ - ^oll 

where Un = {r] : \\r] — i]o\\ < 5„} for any J„ 0. Note that we can also write 
An = ||G;||5„, where ||G;||5„ = supjg^^ By (1A.2[), to show the below 
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bootstrap equicontinuity condition 

G:(m(eo,??) - mo) = Op^dl^ - VoW) m Pl-Probability 



39 



it suffices to show 

(A. 11) lim sup Exw^n < oo. 

?1— >oo 

Note that 

1 n 1 " 

'^n = ^ E(^™ - ^)^X, = -r " Wx, " Px) 

by the condition that the sum of weights Wni^s equals to n. Let = 
{W^i, . . . , PV^„) be exchangeable bootstrap weights generated from Pw', an 
independent copy of Pw- The bootstrap weight conditions Wl and W2 
imply that E\y'W^^ = 1 for i = 1, . . . , n. Let 

m(6'o,r/) - rho 



IV-Vol 



Then we have 



^XW^n 



E°xw sup ||G;m„(?7,ryo)|| 



1 



P^xw sup 
E°xw sup 



< E°xwEw' sup 



5;](VF,,-l)(5x, -Px)m„(r?,r/o) 



i=l 



E(VF,, - Ew'W:,,)i6x, - Px)mn{v,Vo) 



i=l 



^Y.(Wni-W:,,){6x,-Px)mn{v,Vo) 



1=1 



We next further bound E^w^^n by the following symmetrization argu- 
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ment: 



E'^yyAn < E°xw sup 



Wni{5x, - Px)mn{r], 



i=l 



+ E°xwEw' sup 



1 



(A.12) 



< 2E°jcw sup 



\/n . , 



i=l 



We next apply the multiplier inequality in the Appendix, i.e. ()A.9p . with 
Zni = {fe - Px)mn{v,Vo) -ri^Un} to (|Al2]) . Let 



\Zni\\n = sup \\{5x, - Px)mniri,rio) 



We first analyze 



£^xll^ni||n = E°x sup \\{5xi - Px)"i„(r/, || 

< E°x sup ||m„(?/,?7o)(Xi)|| +E°x sup ||£'xm„(r/, r/o)|| 

< 2E°xSn{Xi) 

where Sn is the envelop of the class 5„ defined in d?]). The first inequality 
in the above follows the Fatou's Lemma. Note that (|21|) implies 

(A.13) -^E°x max Sn{Xk) 0, 

l<fc<n 

(A.14) lim sup E°xSn{Xi) < oo, 

n— >oo 

by some analysis, see page 120 of 



211 . Then by (fXlij) , we have lim sup„ E°x\\Zni 



oo. 



The bootstrap conditions W3 and W4 together with the Lemma 4.7 in 
29| imply that £'i4/(maxi<j<„ 0. By the multiplier inequality 

in the Appendix and the above analysis, to prove (|A.lip . it suffices to show 



(A. 15) max Ex sup 

no<i<n ^izu„ 



1 

—p H ("^^j ~ Px)'mn{v,Vo) 



< oo 
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as n — > oo. We will apply the Hoffmann- Jorgensen Inequality (with p = 1) in 

the Appendix to show the moment bound of the above truncated empirical 

processes is finite. First, we establish 
1 



E°x sup 



i=l 



< Ki \ max \\Znk\\n + Gn^{vi)\ 

l_ y TJ l<k<n ) 



(A.16) < h+h, 

where Ki and < t^i < 1 are constants and 
Gn{t) = P°x in 





n 






i=l 


n ' 



Obviously, ()A.13p implies that 1\ — > 0. We next consider 12- Note that the 
assumption SI implies ||G„||5^ = ||^^~^''^ X^iLi ^mlU = 0°p^(X)- Hence we 
can claim that there exists a finite constant Mt such that liminf„ Gn{Mt) > t 
for every 1 > t > 0. Thus we know limsup,„ G^^(i'i) < < oo since 
< ui < 1, and hence (|A.16p < oo. Considering the triangular inequality, 
we have shown (jA.lSP . This completes the whole proof of Lemma [2j □ 

A. 4. Proof of Lemma [7]. The result (I49p is an immediate consequence 
of Lemma 3.4.2 in [Sj]. To show (j5ip . we first apply the symmetrization 
arguments used in the proof of Lemma [21 Thus, for sufficiently small 6, the 
left hand side of (|43p is bounded by 



1 



1=1 



(A.17) 

where VFni's are the assumed bootstrap weights and 

Yni = {{6x, - Px){v{e, ??) - v{e, ??„)) : d{r], r/„) < 6, \\e - OqW < 5}. 

Next the multiplier inequality ()A.9p is employed to further bound (jA.lTp . 
In view of ()A.9p . we only need to figure out the upper bound for 

(A.18) E"x\\Yni\\v, 
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max Ex 

no<i<n 



1 * 



Vs 



for any no > 1 given the assumptions W3 and W4 on the bootstrap weights. 
By similar analysis in the proof of Lemma [2l we know 

where Vn is the envelop function of the class Vs defined in (|46p. The as- 
sumption (|5U|) . together with the analysis of assumption SBl, implies that 



limsup,„ E'^lll^illv^- < oo. Next, the Lemma 3.4.2 in 



34l ] implies that 



E'xl 



:„||v,</f(.,v..L.(P))(i + M|_^) 



By the triangular inequality, we know that (|A.19P has the same upper bound 



as ii^^||Gn 



|v^. This concludes the proof of ()5ip . □ 



A. 5. Proof of Theorem^. By ([38]) and the fact that U{9Q,r]Q) = 0, we 
have 

(A.20) = Li + L2 + 0|,^^.(n"i/2). 

Further, based on the conditions (f35]) and (f37|) . we apply the Lemma 4.2 
in [37[ to obtain that Li = -{U* - [/„)(6'o, + Op^w^^'^^^^ ^ V 
ll^l ~ ''?o||)- By Lemma 3.3.5 in ^] given ([35]) and ([36j) . we have L2 = 



-{Un - f/)(^o,??o) V 11^ - ^oll V 11^1 - 7?o||). By applying CLT 

and Theorem 2.2 in 29[ under the condition (j35p to Li and L2, we have 

(A.21) [/(0^,^^i)-C/(0o,r7o) = O?>,^(n-i/') + o?,^^-(||0^-0o||V||??i-r?o||). 
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We next apply Taylor expansion to U {0, rj~) — U{9q,7]q): 



U{9,fjl)-Ui9o,vo) 

= (1(6 -00, v*~ - m) + oi\\e- eoW v - m\\) 

= U{e - 00, 0) + U{0, r?i - ??o) + o{\\e -OoWv \\fjt - rjoW) 

by the assumed Frechet differentiability of U and linearity of U. Note that U 
has bounded Frechet derivative and U{0, ■) is continuously invertible. Thus, 
we can conclude that 

U(9, rjt) - UiOo, rjo) > c\\rjt - m\\ + 0{\\9 - 0o||) + o{\\e -9o\\\/ Wfjt - rjo\\) 

for some c > 0. Combining the above inequality with (IA.2ip . we can establish 
the below inequality: 

which implies (j39p . □ 

A. 6. Proof of Theorem\^. To prove Theorem [5l we want to show that 

(A.22) P"xw (d{rj*~,rin) > 2*'"(5„ V - ^oll), 6^ G 6, r?i G Hn) 

as n ^ oo and A/„ ^ oo by ()A.2p . The basic idea in proving ()A.22p is first 
to partition the whole parameter space into "shell" s and then we can bound 
the probability of each shell under the conditions (|4ip - (j43p . 

For now we fix M = M„ and then allow it to increase to infinity. We first 
define the shell Sn,j,M as 

Sn,j,M = {{0,7]) eexHn: 2^-^5„ < (i(r?,r/„) < 2^n,div,Vn) > 2^'\\e - OoW} 
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with j ranging over the integers and M > 0. Obviously, the event {9 € 
e, fjt£ Tin - d{f]^, rin) > 2^'{5n V \\6 - 6q\\)} is contained in the union of the 

6 6 
events {(0,r/*) E Snj.A/} for j > M. Thus, we have 

Pxw {difji rjn) > 2^'{6n v\\e-eo\\),eee,fjtenr 



j>M 

< T.Pxw( sup F:{vi9,7^)-v{e,r^n))>o]. 

j>M \iS,^)&S„,j,Aj / 

The second inequahty follows from the definition of r]~. By the smoothness 

6 

condition on v{9,i]), i.e. (j4ip . we have the below inequality when {9,r]) £ 
Sj,n,M for j > M: 

(A.23) Px{v{9,v)-v{9,vn)) ;i - d{i],Vnf + \\9 - 9o\\^ < -2^^-^6l 

for sufficiently large M. 

Considering (jA.23p . we have 

Pxw (rf(?7~, Vn) > 2^'{6n V 11^ - 9o\\),9 G e, G Tin) 

< Y.Pxw( sup MK-Px){v{9,r])-v{9,r]n)) ^ V^2''-^6^, 

j>M \{f,'7)eS„j,A/ 

< Y.Pxw( sup \GU<0,7^) - vi9,r^n))\ ^ V^2^'-'6l] 

j>M \{S,v)eS„,j,M j 

+Px ( sup \Gn{v{9,r^) - v{9,r,n))\ ^ ^2''^-Hl\ 



< 



where the third inequality follows from the Markov inequality and 
Note that the assumption that 5 ^ ipni^)/^"' ^ i^rX^)/^"') is decreas- 
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ing for some < a < 2 implies that ipnicS) < c'^ipni^) for every c > 1. 
Combining another assumption on ipn and ip"^, i.e. ipn{5n) 1^ V^^n ^^"^ 
i^ni^n) < have proved the last inequality in the above. By al- 

lowing M = Mn — > oo, we have completed the proof of (jA.22p . and thus 
Theorem [51 □ 
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